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ABSTRACT 


Multimodal biometric system is a system that is viable in authentication and 
capable of carrying the robustness of the system. Most existing biometric 
systems (ear-fingerprint and face-ear) suffer varying challenges such as large 
variability, high dimensionality, small sample size and average recognition 
time. These lead to the degrading performance and accuracy of the system. 
Sequel to this, multimodal biometric system was developed to overcome those 
challenges. The system was implemented in MATLAB environment. Am 
improved self -organizing feature map was used to classify the fused features 
into known and unknown. The performance of the developed multimodal was 
evaluated based on sensitivity, recognition accuracy and time. 
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I. Introduction 

Multimodal biometric recognition system has drawn the attention of various 
researchers because of the increased high level of security and authentication it 
provides over the existing biometric system. Multimodal recognition system has 
a comparative advantage over the individual biometric system in terms of 
difficulty to hack and manipulate. Multimodal biometrics system combines two 
or more recognition systems into one singular system to overcome the 
limitations of individual biometrics (Ramadan etal., 2015). 

However, different biometrics features or traits collected from different sources 
of same person such as face, ear, fingerprint, palmprint, iris, voiceprint, 
signatures and DNA can be used in developing multimodal biometrics system 
and each biometric trait has its distinctive application, need and benefits. 
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This study considered the ear, fingerprint and face biometric 
system to develop a robust multimodal biometric system 
(face-ear-fingerprints) for maximum accuracy and 
recognition time. The multimodal biometric system 
overcomes challenges encountered by individual biometric 
system such as large variability, high dimensionality, small 
sample size and average recognition time. This studyused an 
improved self-organizing feature as a classifier technique 
and its performance is evaluated using sensitivity, 
recognition accuracy and recognition time. The rest of the 
paper is organized as follows; Section II presents the review 
of the related works. The analysis and methodology 
procedures are described in Section III. Section IV described 
the results and discussion. Conclusion and recommendations 
are described in Section V. 

II. Review of related works 

Biometric recognition system has many publications to its 
credit. Several authors have worked extensively on 
individual biometric system such as iris, ear, face and 
fingerprint. However, despite all these effort there is 
degrading in the performance of the biometric system and 
recognition accuracy. There is need to increased level of 
security and authentication to have a robust biometric 
system. Chang et al. (2003) compared ear recognition with 
face recognition using a standard principal components 
analysis (PCA) technique known as the "eigen-face” 


approach on face and ear images acquired at the University 
of South Florida. The results showed that recognition 
performance was not significantly different for the two 
modalities; with 71.6% and 70.5% accuracies obtained in the 
baseline experiment for ear and face recognition 
respectively. 

Shubhandi and Manohah (2012) proposed an artificial multi¬ 
biometrics approach to fingerprint and face biometrics. The 
study presented an efficient fingerprint and face recognition 
algorithm combining ridge based matching for the 
fingerprint and eigenface approach for the face. The study 
shows improved performance in terms of recognition 
accuracy. Ujwalla et al. (2013) proposed a system with 
combination of Iris, finger print, face and palm geometry. 
Probabilistic Neural Network and Radial Basis Function NN 
classifier were used in identification phase for obtaining 
precision in decision of adaptive and cascade classifier. The 
Back Propagation neural network classifier (BPNN) was 
used in verification phase to classify user as genuine or 
imposter. The result showed 98.8% of accuracy. 

Snehlata etal. [20 14) proposed an implementation of person 
identification fusing face, ear and iris biometric modalities. 
They used PCA based neural network classifier for feature 
extraction from the face and ear images and hamming 
distance for calculating iris templates. The results showed an 
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improvement over the existing result when the modalities 
are combined, but the limitation is there was no separate 
test with non-neutral data and also recognition time was not 
reported. Ismaila et al. [2018) implemented unsupervised 
learning algorithm in multi-modal biometric system that 
made used of palmprint and thumbprint for its suitability. 
The performance of the self-organizing feature map and 
back-propagation neural network was evaluated and 
compared. The back-propagation neural network produced 
recognition accuracy rate of 93.7 while self-organizing 
feature map yielded recognition accuracy rate of 93.5. 

III. METHODOLOGY 

The developed multimodal biometric system was 
implemented using MATLAB R2015a and run on Windows 
10 professional 64-bit operating system, Intel(R) Core[TM) 
i3-2370M CPU @ 2.40GHz, 4GB Random Access Memory and 
500GB hard disk drive with accurate speed for better 
performance of the developed system. The methodology 
involved the acquisition of image datasets, the preprocessing 
of the acquired images, the extraction of features, fusion of 
the features and classification of the extracted features using 
self-feature map. 

Images acquisition 

The face, ear and fingerprints of 120 subjects were captured 
in the same lightening conditions with no illumination 
changes in the size 1200 x 1600 pixels. Six images were 
captured for 120 subjects with total dataset of 2160 images. 
One thousand two hundred and sixty [1260] images were 
used in training the system and the remaining 900 images 
were used to test the system and finally saved in jpeg format. 

Image preprocessing 

Image pre-processing was carried out by converting face, ear 
and fingerprint images into grayscale using histogram 
equalization method. The average face, ear and fingerprint 
vector were calculated and subtracted from the original 
image vectors. This removed noise and other unwanted 
element from the images. Each of the grayscale images are 
expressed and stored in form of matrix in MATLAB which 
eventually converted to vector images for further 
processing. 

Normalization removed any common features that all the 
images shared together, so that each image was left with 
unique features. The common features were discovered by 
finding the average dataset vector of the whole training set 
[face, ear and fingerprint images). Then, the average image 
vector was subtracted from each of the dataset vectors 
which resulted to normalized (face, ear and fingerprint) 
vector using histogram equalization. 

Feature extraction using principal component analysis 

Principal component analysis was used for dimensionality 
reduction by converting the set of correlated images into set 
of uncorrelated eigenvectors. PCA eigenvector method 
considered each pixel in an image as a separate dimension, 
that is, N x N image has N 2 pixels or N 2 dimensions. To 
calculate eigenvector, there was a need to calculate the 
covariance metric for dimensional reduction. The 
eigenvectors were sorted according to their corresponding 
eigenvalues from high to low. The eigenvector 
corresponding to zero, the eigenvalues were discarded while 
those associated with non-zero eigenvalues were kept. 


Feature level Fusion 

The feature set originated from three different sources (face, 
ear and finger) were initially pre-processed and the 
extracted features from each of the dataset formed a feature 
vector. These features from each of the dataset were then 
concatenated to form a new feature vector. Concatenation of 
feature set increased the dimensionality of the fused feature 
vector. Feature level fusion was used in this study because it 
is can fuse incompatible feature vectors from multiple 
modalities. 

Training and testing of Multimodal Biometric System 
using SOFM 

The input vectors were presented to the network based on 
the initial weights that was chosen at random and the 
neuron with weights closest to the input image vector was 
declared as the winner. Then, weights of all of the neurons in 
the neighborhood of the winning neuron were adjusted by 
an amount inversely proportional to the Euclidean distance. 

During the training phase, the data acquisition, pre¬ 
processing, feature extraction process and feature 
concatenation process took place concurrently and the fused 
image sample was presented to the SOFM at a stretch. For 
each node, the number of '‘wins’’ was recorded along with 
the label of the input sample. The weight vectors for the 
nodes were updated as described in the learning phase. By 
the end of the stage, each node of the SOFM recorded two 
values, that is, it generated the total number of winning 
times for "known fused images" in image database, and the 
total number of winning times for "unknown fused images” 
in image database. 

During the testing phase, SOFM classified individual image 
based on correctly or incorrectly identified images. The fused 
input vector was compared with all nodes of the SOFM and 
the best match found based on minimum Euclidean 
distance and was further subjected to some selected 
threshold values such as 0.24, 0.35, 0.47 and 0.58. The 
final output of the classified image was displayed as known or 
unknown and the results were recorded as illustrated in 
Figure 1. 

The steps for Self-Organizing Feature Map algorithm 
considered in this study were as follows; 

1. Each node's weights was initialized. 

2. A vector was chosen at random from the set of training 
data and presented to the network. 

3. Every node in the network was examined to calculate 
which ones' weights are most like the input vector. The 
winning node is commonly known as the Best Matching 
Unit (BMU). 

4. The radius of the neighborhood of the BMU was 
calculated. This value started large. Typically it was set to 
be the radius of the network, diminishing each time-step. 

5. Any nodes found within the radius of the BMU, calculated 
in (iv.), was adjusted to make them more like the input 
vector (Equation 2.17 and Equation 2.18). The closer a 
node is to the BMU, the more its' weights are altered 
(Equation 2.19). 

6. Repeat (ii) for N iterations. 
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Figure l:Flowchart showing trained and tested of selected images with Self Organizing Feature Map 


IV. RESULTS AND DISCUSSION 

The performance of multimodal system with respect to 
SOFM was evaluated based on false positive rate, sensitivity, 
specificity, recognition accuracy and recognition time. The 
accuracy generated by SOFM was analyzed by using ear- 
fingerprint, face-fingerprints, face-ear and face-ear- 
fingerprints dataset features at different threshold values of 
0.24, 0.35, 0.47 and 0.58 as illustrated in Tables 1-4. The 
results obtained at different threshold values showed that 
SOFM perform better at 0.58 threshold value than the rest of 
the values. The results of the experiments with SOFM were 
presented in Tablel, Table 2, Table 3 and Table 4 based on 


false positive rate, sensitivity, specificity, recognition 
accuracy and recognition time. 

The recognition accuracy at multimodal level with SOFM 
generated 94% of 0.24,95% of 0.35,96% of 0.47 and 97% of 
0.58. The average recognition time produced at multimodal 
level are 97.05s at 0.24, 92.10s at 0.35, 92.05 s at 0.47 and 
91.60s at 0.58. In terms of false positive rate, SOFM 
generated 8 at 0.24, 8 at 0.35, 6 at 0.47 and 4 at 0.58, while 
sensitivity and specificity were on increase when subjected 
to threshold values of 0.24, 0.35, 0.47 and 0.58. 
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Table 1: EAR-FINGERPRINT 


Threshold 

TP FP FN TN 

FPR (%) 

Sensitivity 

(%) 

Specificity (%) 

Accuracy 

(%) 

Recognition 
Time (Sec) 

0.24 

50 

5 

0 

45 

10 

100.00 

90.00 

95.00 

98.20 

0.35 

49 

5 

1 

45 

10 

98.00 

90.00 

94.00 

92.85 

0.47 

48 

3 

2 

47 

6 

96.00 

94.00 

95.00 

92.20 

0.58 

47 

3 

3 

47 

6 

94.00 

94.00 

94.00 

91.00 


Table 2: FACE-FINGERPRINT 


Threshold 

TP FP FN TN 

FPR 

(%) 

Sensitivity 

(%) 

Specificity (%) 

Accuracy 

(%) 

Recognition Time 
(Sec) 

0.24 

48 

7 

2 

43 

14 

96.00 

86.00 

91.00 

95.05 

0.35 

48 

5 

2 

45 

10 

96.00 

90.00 

93.00 

92.90 

0.47 

48 

3 

2 

47 

6 

96.00 

94.00 

95.00 

91.15 

0.58 

47 

3 

3 

47 

6 

94.00 

94.00 

94.00 

91.30 


Table 3: FACE-EAR 


Threshold 

TP FP FN TN 

FPR 

(%) 

Sensitivity 

(%) 

Specificity (%) 

Accuracy 

(%) 

Recognition Time 
(Sec) 

0.24 

48 

7 

2 

43 

14 

96.00 

86.00 

91.00 

96.65 

0.35 

47 

6 

3 

44 

12 

94.00 

88.00 

91.00 

91.65 

0.47 

46 

4 

4 

46 

8 

92.00 

92.00 

92.00 

91.65 

0.58 

46 

5 

4 

45 

10 

92.00 

90.00 

91.00 

91.35 


Table 4: FACE-EAR-FINGERPRINT (MULTIMODAL LEVEL] 


Threshold 

TP FP FN TN 

FPR 

(%] 

Sensitivity 

(%) 

Specificity (%) 

Accuracy 

(%) 

Recognition Time 
(Sec) 

0.24 

50 

4 

0 

46 

8 

100.00 

92.00 

94.00 

97.05 

0.35 

49 

4 

1 

46 

8 

98.00 

92.00 

95.00 

92.10 

0.47 

48 

3 

2 

48 

6 

98.00 

94.00 

96.00 

92.05 

0.58 

49 

2 

3 

48 

4 

96.00 

96.00 

97.00 

91.60 


V. CONCLUSION AND RECOMMENDATION 

Am improved SOFM classified faster in terms of recognition 
accuracy, recognition time, specificity and sensitivity for 
multimodal biometric systems over the existing techniques 
due to the fact that the images acquired were concatenated 
into a single fused vector making it easier for classification. 
This has contributed greatly in the areas of e-passports, 
border control and e-learning. Future works can be carried 
out by comparing and evaluate the performance of other 
neural network techniques. 
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